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ABSTRACT 


Many  researchers  are  attempting  to  quantify  or 
understand  the  value  of  information,  especially  for  the 
Army  as  it  enters  its  transformation.  Information  can  be 
decomposed  into  various  qualities.  Three  of  these 
qualities,  timeliness,  accuracy,  and  completeness,  form  the 
basis  for  this  thesis.  This  thesis  uses  a  simulation 
framework  developed  by  the  author  to  analyze  the  three 
components  of  information  listed  above.  The  scenario 
selected  is  a  typical  vignette  of  an  Objective  Force 
company-sized  element  conducting  offensive  operations 
against  threat  elements.  Knowledge  of  the  threat  was 
compromised  by  the  presence  of  decoy  elements  as  well  as 
previously  damaged  or  killed  systems  (BDA) .  In  this 
scenario  the  fires  are  initiated  from  standoff  ranges.  The 
initial  and  running  assessments  of  the  threat  composition 
are  made  based  on  the  information  provided  by  sensors  on 
board  the  unit's  organic  unmanned  aerial  vehicles  (UAVs) . 
Analysis  of  the  simulation  results  helps  in  understanding 
how  components  of  information  quality  affect  the  overall 
effectiveness  of  the  force  as  reflected  in  an  efficiency 
measure.  Additionally,  critical  thresholds  for  timeliness, 
accuracy,  and  completeness  of  information  are  pinpointed  to 
inform  Objective  Force  decision  makers. 
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EXECUTIVE  SUMMARY 


The  US  Army  and  its  Training  and  Doctrine  Command  have 
taken  on  the  responsibility  for  determining  how  to  trade 
heavy  caliber  weapons  and  heavy  armor  for  lighter,  more 
agile,  and  information-dependent  fighting  systems. 
Understanding  the  impact  of  how  information  enables  the 
future  Objective  Force  is  critical  in  this  high-risk 
endeavor.  A  common  criticism  of  information-based  warfare 
is,  "You  can't  know  an  enemy  to  death." 

There  is  a  difference  between  the  value  and  the 
quality  of  information.  Value  is  subjective  and  depends  on 
the  decision  maker  as  well  as  his  information  needs. 
Battle  command  analysis  focuses  on  information  value  to 
different  levels  of  information  users.  Information  quality 
is  more  objective  and  is  the  focus  of  this  research. 

Information  quality  is  defined  by  its  timeliness, 
accuracy  and  completeness.1  The  overarching  problem 
examined  by  this  research  is  determining  the  relative 
influence  these  components  of  information  quality  have  on 
combat  outcomes. 

The  methodology  used  to  explore  this  problem  consists 
of  two  major  pieces.  The  first  step  is  the  development  of 
a  simulation  framework  that  models  an  Objective  Force  unit 
engaged  in  combat  operations.  A  Mounted  Combat  System 
(MCS)  Company,  one  of  the  Unit  of  Action  (UA)  sub-elements, 
is  the  subject  of  the  simulation  tool.  The  MCS  Company  is 
optimized  for  extended  line  of  sight  (LOS)  with  beyond  line 
of  sight  (BLOS)  fires,  and  employs  chemical  energy  (CE)  and 

1  perry,  p.  30 
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kinetic  energy  (KE)  munitions  to  engage  at  standoff.2  Its 
mission  in  the  simulation  model  is  to  identify  and 
eliminate  enemy  targets  dispersed  throughout  an  objective 
area  using  organic  fires  at  standoff  ranges.  The  future 
threat,  recognizing  its  overmatch  by  Objective  Force  units, 
will  use  adaptive  tactics,  deception,  and  physical  decoys 
to  their  own  advantage.  The  simulation  tool  mirrors  this 
operational  environment  with  its  ability  to  model  decoys, 
stationary  and  moving  live  targets,  and  battlefield  clutter 
in  the  form  of  battle  damaged  vehicles.  The  simulation 
also  models  the  three  organic  Unmanned  Aerial  Vehicles  (UAV 
CL  II)  that  are  used  to  provide  the  BDA  and  target  location 
data . 

The  second  step  is  performing  statistical  analysis  on 
the  simulation  output.  This  facilitates  making 
observations  about  information  quality  component 
relationships  and  how  they  impact  force  effectiveness  as 
reflected  in  an  efficiency  measure.  A  33  full  factorial 
designed  experiment  is  used  to  structure  the  simulation 
responses  by  looking  at  timeliness,  accuracy,  and 
completeness  each  at  three  levels.  The  response  for  each 
design  point  of  the  experiment  is  the  number  of  rounds 
required  to  eliminate  a  pre-determined  percentage  of  enemy 
targets.  Analysis  of  variance,  polynomial  regression  and 
data  correlation  are  used  to  make  broad  observations  about 
the  dynamics  of  these  information  quality  components. 

The  results  of  this  study  show  that  timeliness, 
accuracy,  and  completeness  are  significant  in  influencing 
the  measure  of  effectiveness,  but  there  is  a  difference  in 
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their  relative  importance  with  regard  to  how  much  of  the 
variability  in  the  response  each  component  can  explain.  In 
this  scenario,  the  completeness  factor  explains  31%  of  the 
variability  while  accuracy  and  timeliness  explain  23%  and 
12%  respectively.  Completeness  stands  out  with  respect  to 
the  importance  of  a  single  factor. 

Additionally,  some  components  produce  accelerated 
changes  in  the  MOE  compared  to  the  degree  of  change  in  the 
level  of  the  factor.  Increasing  the  timeliness  factor 
level  from  medium  to  high  results  in  tremendous 

improvements  in  efficiency  while  a  change  from  medium  to 
low  results  in  only  a  small  decrease  in  efficiency. 

However,  increasing  the  completeness  factor  level  from 
medium  to  high  results  in  a  mild  increase  in  efficiency 
while  a  change  from  medium  to  low  results  in  a 
significantly  large  decrease  in  efficiency. 

Finally,  there  is  a  synergistic  effect  when  the 
combination  of  timeliness  and  accuracy  are  held  at  their 
high  levels.  Three  of  the  top  five  simulation  run 

responses  occur  when  this  is  the  case.  Timeliness  and 

accuracy  combine  in  a  way  that  is  resilient  to  the  effect 
of  completeness. 

These  dynamics  are  certainly  scenario  specific,  but 
this  study  demonstrates  that  they  do  exist  and  provides  a 
methodology  and  framework  with  which  to  discover  them. 
This  information  in  the  hands  of  a  concept  developer  allows 
him  to  make  wise  choices  in  determining  what  technologies 
and  tactics  are  needed  to  improve  the  success  of  units 
optimized  for  specific  missions. 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


I. 


INTRODUCTION 


A.  BACKGROUND 

The  U.S.  Army's  new  Objective  Force  design  calls  for  a 
new  paradigm  in  fighting  our  future  battles.  Objective 
Force  units  are  anticipated  to  have  the  capability  to  "see 
first,  understand  first,  act  first  and  finish  decisively."3 4 
The  key  to  making  this  concept  a  reality  is  an  overwhelming 
situational  understanding  largely  made  possible  by  the 
ability  to  obtain,  process  and  rapidly  move  an  abundance  of 
information  on  the  future  battlefield. 

The  traditional  elements  of  combat  power  include 
Maneuver,  Firepower,  Protection  and  Leadership  resulting  in 
the  formula:  CP=M+F+P+L.  According  to  Army  concept 
developers,  however,  it  is  envisioned  that  in  Objective 
Force  units  a  "situational  understanding  derived  from  real¬ 
time,  accurate  Information  raises  combat  power 
exponentially:  CP=  (M+F+P  +  L)  Information  _  „4 

One  example  of  the  impact  of  information,  consistent 
with  the  idea  expressed  in  the  above  formula,  is  an 
observation  by  VADM(ret.)  Cebrowski  in  Transformation 
Trends:  "The  air  force  says  that  a  target  once  requiring 
1,000  bombs  to  destroy  now  requires  only  one.  That 
magnitude  of  change  is  owed  almost  entirely  to  information 
technology  and  processes."5 


3  TRADOC  Pamphlet  525-3-90/O&O,  p.  4-3 

4  TRADOC  Pamphlet  525-3-90/O&O,  p.  3-1 

5  Cebrowski,  p.  2 
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B .  PROBLEM 

In  this  day  and  age  there  is  a  lot  of  emphasis  on  the 
merits  of  information,  and  much  effort  is  going  into  how  it 
can  be  obtained  more  quickly,  completely  and  accurately. 
However,  "little  has  been  done  to  establish  a  clear 
relationship  between  information  and  the  outcome  of 
military  operations."6  The  first  step  in  attempting  to 
discern  what  this  relationship  looks  like  is  defining  what 
is  meant  by  the  term  information. 

According  to  Dr.  Walter  L.  Perry  in  his  article, 
"Knowledge  and  Combat  Outcomes,"  information  has  two  main 
attributes:  value  and  quality.7  Information  has  value  if  it 
informs  the  commander  and  answers  questions  posed  by  his 
intelligence  requirements  such  as  Priority  Intelligence 
Requirements  (PIR)  or  Commanders  Critical  Information 
Requirements  (CCIR) .  In  other  words,  valuable  information 
is  relevant  to  the  situation  at  hand. 

The  quality  of  information,  however,  depends  on  its 
accuracy,  timeliness  and  completeness.8  Valuable 
information  may  not  always  be  of  high  quality.  On  the 
other  hand,  information  could  have  high  quality  but  have  no 
relevance  to  the  situation  at  hand,  and  therefore  have 
little  to  no  value. 

The  purpose  of  this  thesis  is  to  investigate  the 
impact  of  information  on  Objective  Force  operations.  The 
focus  will  be  on  information  quality,  as  defined  by  Perry 
above,  and  the  goal  is  to  draw  some  broad  conclusions  about 

6  Darilek,  p.  2 

7  perry,  p.  30 

8  perry,  p.  30 
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how  the  individual  components  of  information  quality  can 
influence  combat  outcomes. 

A  specific  analysis  of  information  timeliness. 


accuracy , 

and 

completeness  and 

their  impact  on  a 

combat 

scenario 

does 

not  yet 

exist . 

These 

three  terms 

were 

discussed 

in 

a  Joint 

Battle 

Damage 

Assessment 

Joint 

Feasibility  Study  Report  in  September  2000. 9  However,  in 
this  report  they  were  used  as  measures  of  effectiveness  to 
evaluate  mobile  target  vs.  fixed  target  battle  damage 
assessment  (BDA)  processes,  not  inputs  to  the  problem. 

C .  METHODOLOGY 

Under  the  Objective  Force  concept,  the  Unit  of  Action 
(UA)  takes  on  a  role  similar  to  that  of  the  traditional 
maneuver  brigade.  There  are  many  critical  tasks  that  must 
be  done  with  a  high  level  of  precision  by  the  UA,  such  as 
firing  and  maneuvering  under  contact,  delivering  fires  at 
standoff,  and  assuring  mobility  near  the  objective.10  An 
additional  critical  task  is  tracking  and  evaluating  Battle 
Damage  Assessment  (BDA) .  Accurate  BDA  facilitates  at  least 
two  things:  (1)  decisive  action  by  the  commander  so  he 
knows  when  he  can  transition  to  subsequent  actions  and 
maintain  pressure  on  the  enemy,  and  (2)  efficient 
expenditure  of  limited  munitions.11 

For  this  thesis,  a  simulation  model  called  the  Mounted 
Combat  System  Killing  Machine  (MCSKM)  was  developed.  The 
simulation  treats  BDA,  target  type  and  target  location  as 
the  types  of  information  under  observation.  With  a  focus 

9  Joint  Battle  Damage  Assessment  Feasibility  Report,  p.  2-3 

10  TRADOC  Pamphlet  525-3-90/O&O,  p.  4-4 

11  TRADOC  Pamphlet  525-3-90/O&O,  p.  4-13 
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on  efficient  expenditure  of  munitions,  the  model  measures 
the  impact  of  this  information  gain  on  the  results  of  a 
specific  scenario. 

A  Mounted  Combat  System  (MCS)  Company,  one  of  the  UA 
sub-elements,  is  the  subject  of  the  simulation.  It  is 
optimized  for  extended  line  of  sight  (LOS)  with  beyond  line 
of  sight  (BLOS)  fires,  and  employs  chemical  energy  (CE)  and 
kinetic  energy  (KE)  munitions  to  engage  at  standoff.12  Its 
mission  in  the  simulation  model  is  to  identify  and 
eliminate  targets  dispersed  throughout  an  objective  area 
using  organic  fires  at  standoff  ranges.  The  MCS  Company 
has  a  total  of  10  MCS  weapon  platforms  available  to  engage 
targets  as  shown  in  figure  1  below.  This  unit  also  has 
three  organic  Unmanned  Aerial  Vehicles  (UAV  CL  II)  that  are 
part  of  its  table  of  organization  and  equipment  (TOE)  as 
shown  in  figure  2  below.  These  UAVs  are  used  to  provide 
the  BDA  and  target  location  data. 


12  TRADOC  Pamphlet  525-3-90/O&O,  p.  3-23 
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Figure  1 . 


MCS  Company  Equipment 
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Mounted  Combat  System  Company 

Mounted  Combat  System  Company  X  2 
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10  Mounted  Combat  Systems  (MCS) 
1  C2V 

3  ARV-RSTAs  (6  Ton  Variant) 
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3  UAV  CL  II  L/C  Units 
3  UAV  CL  II  Aerial  Vehicles 
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Figure  2.  MCS  Company  Organization 


The  components  of  timeliness,  accuracy  and 
completeness  of  the  information  captured  on  the  ground  are 
varied  and  the  resulting  impact  on  the  number  of  munitions 
it  takes  to  eliminate  a  certain  percentage  of  the  live 
targets  is  measured.  Listed  below  are  the  definitions  of 
these  information  quality  components  and  how  they  are 
represented  in  the  MCSKM. 

1.  Timeliness 

Timeliness  reflects  the  relationship  between  the  age 
of  an  information  item  and  the  tasks  or  missions  it  must 
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support.13  In  the  MCSKM,  the  timeliness  factor  represents 
the  amount  of  time  it  takes  from  the  detection  of  a  target 
to  the  impact  of  a  round  on  the  target.  The  processes 
imbedded  in  this  factor  are  the  UAV  data  transmission  time, 
man/machine  image  processing  time,  firing  decision  time  and 
the  round  time  of  flight.  Essentially  this  is  the  time  it 
takes  for  raw  data  to  become  actionable  information 
combined  with  the  time  to  complete  the  resulting  action. 

2 .  Accuracy 

Accuracy  is  a  measure  of  how  faithfully  the  items  of 
information  represent  the  realities  they  describe.14  In  the 
MCSKM,  accuracy  is  represented  by  the  conditional 
probability  of  classification  given  that  one  of  the  three 
battlefield  entities  is  present.  This  is  the  probability 
that  a  live  target,  dead  target  or  decoy  will  be  classified 
as  such  given  that  it  was  detected.  In  the  MCSKM,  if  an 
entity  is  present  in  the  area  being  searched  it  will  be 
detected  with  a  probability  of  1.0  for  the  sake  of 
simplicity.  Therefore,  accuracy  is  purely  a  function  of 
the  quality  of  the  classification  process. 

3.  Completeness 

Completeness  describes  the  level  to  which  all  the 
relevant  items  of  information  are  available  including 
entities  (such  as  targets) ,  attributes  (such  as  movement) 
and  the  relationships  between  them.15  In  the  MCSKM,  the 
amount  of  area  on  the  ground  a  UAV  can  observe  and  evaluate 
for  the  presence  of  targets  in  a  given  unit  of  time 
represents  completeness  of  information. 


13 

Alberts, 

P- 

85 

14 

Alberts, 

P- 

84 

15 

Alberts, 

P- 
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II.  MODEL  DEVELOPMENT 


A.  SCENARIO 

The  objective  area  for  the  MCSKM  scenario  is  an  8 
kilometer  by  8  kilometer  box  of  primarily  open,  rolling 
terrain.  The  MCS  Company  is  located  in  an  attack  by  fire 
position  and,  with  standoff  range  firing  capability, 
destroys  targets  in  the  targeted  area  of  interest  (TAI)  in 
support  of  a  follow-on  assault  by  an  adjacent  infantry 
company . 


8  km 


Figure  3.  Scenario  Environment 

Targets  are  randomly  and  uniformly  dispersed 
throughout  the  objective  area.  There  are  50  total  targets 
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and  they  are  broken  down  into  three  types  with  the 
following  distribution:  36  live,  7  dead  and  7  decoy.  Half 
of  the  live  targets  are  specified  as  movers  and  will  move 
randomly  until  killed  by  a  munition  fired  from  an  MCS 
weapon  platform.  Stationary  targets  represent  systems 

conducting  a  static  defense,  command  posts,  air  defense 
assets,  or  other  fixed  sites.  Dead  targets  are  systems 
that  are  previously  damaged  or  killed.  Decoys  are  non¬ 
moving  entities  that  have  no  military  significance  but  can 
be  mistaken  for  valid,  live  targets.  See  table  1  for 
target  summary: 

Table  1.  Distribution  of  Target  Types 


Target  Types 

Number 

Live  (moving) 

18 

Live  (stationary) 

18 

Dead 

7 

Decoy 

7 

Three  organic  UAVs  fly  in  a  random  pattern  and  report 
perceived  target  imagery  to  the  analysts  in  the  command 
post.  This  target  imagery  serves  as  the  sole  basis  for 
target  location  and  target  type.  With  this  information  a 
decision  will  be  made  to  fire  or  not  fire  at  a  target. 
There  are  no  other  reconnaissance  assets  in  the  area  of 
influence  except  what  gets  transmitted  via  the  UAVs. 

If  a  target  is  perceived  as  live  then  a  decision  to 
fire  at  that  target  is  made.  The  end  state  is  achieved 
when  80%  of  the  live  targets  are  destroyed.  It  is 
important  to  note  that  the  values  chosen  for  this  scenario 
are  easily  modified  by  the  user  in  order  to  explore  other 
scenarios . 
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B .  GENERAL  MODEL 

The  Mounted  Combat  System  Killing  Machine  (MCSKM)  is  a 
discrete-event  simulation  written  in  the  JAVA  programming 
language.  The  MCSKM  implements  the  simulation  tool  Simkit, 
a  discrete-event  simulation  package  developed  and 
maintained  by  Research  Assistant  Professor  Arnold  H.  Buss 
of  the  Naval  Postgraduate  School.  The  objective  of  the 
MCSKM  is  to  provide  a  framework  to  explore  the  information 
quality  components  of  timeliness,  accuracy  and  completeness 
and  how  these  factors  influence  the  number  of  munitions 
required  to  kill  a  certain  percentage  of  the  targets. 
Table  2  below  shows  all  of  the  available  parameter 
adjustments  that  can  be  made  by  the  user.  Experimentation 
was  done  with  all  of  these  settings  in  determining  the 
right  mix  for  the  final  experiment.  These  excursions  will 
be  discussed  in  Chapter  IV.  For  the  analysis  in  this 
thesis,  the  bold  settings  remained  fixed  while  the 
remaining  settings  were  varied  in  the  experiment  described 
in  Chapter  III . 
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Table  2  . 


MCSKM  Available  Adjustable  Parameters 


Simulation  Factors 

Description 

Good  Value 

Medium  Value 

Low  Value 

probability  of  detection 

probability  of  detecting  a  target  (live,  dead  or 
decoy) 

1.0 

probability  of  false  detection 

probability  of  detecting  a  live  target  when  no 
targets  of  any  type  are  present  (type  II  error) 

0.0 

probability  of  determination 
(perceived  |  actual) 

probability  of  determining  what  a  target  is  once 
detected 

o 

(0 

p(live  |  live) 

0.8 

0.6 

0.4 

□ 

p(dead  |  live) 

0.1 

0.2 

0.3 

o 

o 

p(decoy  |  live) 

0.1 

0.2 

0.3 

< 

p(live  |  dead) 

0.1 

0.2 

0.3 

p(dead  |  dead) 

0.8 

0.6 

0.4 

p(decoy  |  dead) 

0.1 

0.2 

0.3 

p(live  |  decoy) 

0.1 

0.2 

0.3 

p(dead  |  decoy) 

0.1 

0.2 

0.3 

p(decoy  |  decoy) 

0.8 

0.6 

0.6 

'imeli- 

ness 

processing  time 

process  =  UAV  glimpse  trasmission  time  + 
man/machine  image  processing  time  + 
firing  decision  time  +  round  time  of  flight 

10  sec 

30  sec 

60  sec 

i— 

UAV  transmission  interval 

seconds  between  UAV  transmissions 

5 

(/> 

to 

a> 

c 

<D 

4-> 

Number  of  UAVs 

How  many  simultaneous  randomly  searching 
UAVs  in  the  air  -  should  be  three  based  on  the 

Nov  02  FCS  O&O 

3 

a> 

a 

E 

box  size 

how  much  area  a  UAV  can  see  in  a  singe 
glimpse  (single  dimension  of  box  in  meters) 

400 

200 

100 

o 

o 

UAV  speed 

km/hr 

120 

60 

30 

bold  =  held  fixed  In  simulation 


The  MCSKM  is  comprised  of  two  basic  processes:  a  UAV 
process  and  a  shooting  process.  Both  of  these  processes 
will  be  explained  in  detail  in  the  next  section.  In 
general,  a  UAV  process  is  instantiated  (i.e.,  created  in 
the  software)  for  each  UAV  represented  in  the  model.  In 
this  model  there  are  three  UAV  processes  in  place.  There 
is  only  one  shooting  process  in  place  and  it  "listens"  to 
each  UAV  process  in  order  to  track  individual  UAV  movements 
and  locations.  Figure  4  below  displays  this  listening 
relationship . 
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Figure  4.  MCSKM  Listening  Scheme 

While  each  UAV  process  controls  the  UAV  movement,  the 
shooting  process  does  all  of  the  real  work  in  the  model. 
The  shooting  process  manages  all  target  movements,  target 
classifications,  target  state  changes,  firing  delays  and 
kill  adjudications. 

C.  INITIALIZATION 

At  the  beginning  of  each  run  of  the  MCSKM,  all  targets 
are  given  an  exact  grid  location  based  on  the  8  kilometers 
by  8  kilometers  objective  area.  These  locations  are 
random,  uniformly  distributed  and  given  in  terms  of  meters. 
For  example,  the  lower  left  corner  of  the  objective  area 
would  be  grid  location  (0.0,  0.0)  and  a  target  that  is  5 

kilometers  to  the  right  of  the  origin  and  3  kilometers  up 
would  be  at  grid  location  (5000,  3000) .  For  the  50%  of  the 
live  targets  that  are  designated  as  movers,  they  are 
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assigned  an  initial  random  azimuth  [0,2tf]  to  begin  movement 
as  well. 


The  UAV  locations  are  represented  differently  from  the 
targets.  Based  on  the  size  of  the  box  representing  how 
much  area  a  UAV  can  see  in  a  single  glimpse,  the  objective 
area  is  divided  up  into  grids  of  the  same  dimension.  For 
example,  if  the  box  size  representing  how  much  a  UAV  can 
monitor  on  the  ground  is  400  meters  by  400  meters  for  a 
given  run,  then  the  objective  area  is  divided  into  a  20  by 
20  grid  system  (8000m/400m  =  20,  the  number  of  grids  on 
each  axis) .  Movement  will  be  described  later,  but  each  UAV 
will  have  a  random  starting  location  in  one  of  these  grids 
for  each  run  of  the  MCSKM.  Figure  5  below  demonstrates 
starting  locations  of  (5,  5),  (10,  15)  and  (18,  10)  for  the 
three  UAVS  as  an  example. 


Figure  5.  UAV  Starting  Locations  in  AT 
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D .  UAV  PROCESS 

Movement  of  a  single  UAV  is  simulated  by  "looking"  at 
one  particular  grid  square  for  the  amount  of  time  it  would 
take  the  UAV  to  move  across  the  grid  square  in  a  linear 
fashion  at  a  fixed  speed.  For  example,  if  UAV  speed  =  120 
km/h  and  the  grid  square  is  400m  x  400m  then  the  time  in 
grid  =  (400m)/ (120  km/h)  =  12  seconds.  The  choice  to  move 
from  one  grid  square  to  another  instead  of  tracing  out  a 
precise  path  along  exact  coordinates  was  made  for  the  sake 
of  simplicity  in  programming.  To  travel  400m  in  12  seconds 
with  a  sensor  sweep  width  of  400m  is  roughly  equivalent  to 
occupying  a  400m  by  400m  grid  square  for  12  seconds. 
Although  some  precision  is  lost  in  the  case  of  a  diagonal 
move,  it  is  not  a  great  concern  in  light  of  the  fact  that 
the  UAV  movement  is  already  abstracted. 

After  this  time  has  passed  the  UAV  "moves"  to  an 
adjacent  grid  square  in  a  random  manner.  The  UAV  can  move 
into  any  one  of  the  eight  adjacent  grid  squares  but  it 
cannot  remain  stationary.  If  the  UAV  is  on  the  border  of 
the  AI  it  is  not  allowed  to  move  in  any  direction  that 
would  take  it  outside  of  the  AI .  The  footprint  of  what  the 
UAV  can  see  (have  the  potential  to  detect  and  classify)  on 
the  ground  is  represented  by  the  size  of  the  grid  square. 

This  is  the  part  of  the  model  where  completeness  plays 
its  role.  The  size  of  the  grid  square  and  the  speed  of  the 
UAV,  in  conjunction  with  each  other,  control  the  amount  of 
information  available  per  period  of  time.  With  12  seconds 
time  on  station  for  any  size  grid  square,  the  completeness 
levels  are  determined  by  the  following  parameter  value 
combinations : 
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Table  3 . 


Completeness  Levels 


High 

Medium 

Low 

UAV  Speed 

120  km/h 

60  km/h 

30  km/h 

Grid  Size 

400m 

200m 

100m 

Figure  6  below  is  an  event  graph  of  the  UAV  Process. 
The  RUN  event  initializes  the  UAV  in  a  random  starting 
location  and  schedules  the  first  arrival  in  a  grid  square. 
Upon  arrival,  the  "UAV  Depart  Grid"  is  scheduled  for  when 
the  time  in  the  grid  will  have  elapsed  and  a  "Determine 
Targets"  is  scheduled  immediately  which  is  the  UAV' s  first 
attempt  to  detect  and  classify  any  targets  present.  The 
UAV  process  only  signals  for  the  "Determine  Targets"  event 
to  happen;  the  actual  work  of  this  event  is  done  in  the 
shooting  process  and  will  be  explained  later.  After 
arrival  in  the  grid  and  taking  an  initial  glimpse,  the  UAV 
will  continue  to  attempt  to  determine  targets  by  taking  a 
glimpse  at  5-second  intervals  until  its  time  in  the  grid 
has  expired.  Once  the  time  is  up  it  will  move  to  another 
grid  in  the  manner  explained  above. 
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UAV  Process 


Parameters  State  Variables 

t  -  time  in  grid  gridl  -  grid  number  in  X  orientation 

t;  -  time  between  glimpses  gridJ  -  grid  number  in  Y  orientation 

arriveTime  -  time  of  arrival  in  grid 


Figure  6. 


UAV  Process  Event  Graph 


E .  SHOOTING  PROCESS 

The  Shooting  Process  is  initiated  by  any  UAV 
Process's  call  for  a  "Determine  Targets"  event.  The  model 
contains  variables  for  the  probability  of  false  detection 
(type  II  error) ,  the  probability  of  detection,  and  the 
conditional  probability  of  classification  given  a 
detection.  However,  the  probability  of  false  detection  was 
fixed  at  0.0  and  the  probability  of  detection  was  fixed  at 
1.0  for  the  sake  of  simplicity  in  this  implementation  of 
the  MCSKM.  The  reasoning  for  these  choices  is  explained  in 
chapter  IV.  Therefore,  if  a  target  is  present  it  will  be 
detected.  Once  detection  occurs,  the  UAV  will  classify  the 
target  based  on  the  conditional  probability  of 
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classification  parameter  setting.  If  a  target  is  detected 
and  classified  as  live,  then  a  decision  to  fire  is  made. 

The  impact  of  the  round  will  be  delayed  by  a  number  of 
seconds  based  on  the  processing  time  parameter.  This 
simulates  the  time  it  takes  for  raw  data  to  become 
actionable  information  and  then  be  acted  upon.  Once  a 
target  is  identified  as  live  and  has  a  round  fired  at  it, 
that  target  is  not  eligible  for  detection  again  until  that 
round  has  landed.  This  prevents  multiple  rounds  being  fired 
at  the  same  live  target  in  a  single  grid  square.  Since 
half  of  the  live  targets  are  moving,  there  is  always  a 
chance  that  the  original  target  may  not  be  in  the  same  grid 
when  the  round  makes  impact. 

This  process  keeps  iterating  until  a  specified  target 
attrition  level  is  achieved.  This  level  is  variable  in  the 
MCSKM  but  for  this  analysis  the  attrition  level  is  80%  of 
the  instantiated  live  targets  as  requested  by  the  scenario 
under  observation.  The  implications  of  changing  this 
threshold  are  discussed  in  chapter  IV.  The  simulation 
terminates  once  that  attrition  level  is  met.  The  measure 
of  effectiveness  for  a  given  run  is  the  number  of  munitions 
required  to  reach  the  specified  level  of  attrition. 

Figure  7  below  is  an  event  graph  of  the  Shooting 
Process.  Since  this  is  where  the  bulk  of  the  simulation 
takes  place,  each  event  will  be  discussed  in  detail. 
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Shooting  Process 


(target  is  perceived  live  in  grid  I,J) 


(target  hit  in  grid  I,J  was  live) 


{update  target  moves, 
determine  candidate  list, 

ID  any  target  perceived  live} 


{update  target  moves, 
determine  candidate  list, 
check  for  original  target, 
check  if  target  was  live} 


{set  targetType(dead), 
set  targetMoving(false), 
liveTargets-} 


Parameters 

tp  -  time  to  process  data 
Pc  -  p(perceived|actual) 


State  Variables 

munitions  -  number  of  MCS  munitions  fired 
liveTargets  -  number  of  live  targets  remaining 


Figure  7.  Shooting  Process  Event  Graph 


1.  "Determine  Targets"  Event 

The  "Determine  Targets"  event  in  the  Shooting  Process 
is  scheduled  by  the  "Determine  Targets"  event  in  the  UAV 
Process.  The  Shooting  Process  knows  when  to  conduct  this 
event  because  it  "listens"  to  the  UAV  Processes.  The  UAV 
Process  passes  its  grid  location  so  that  the  Shooting 
Process  knows  where  to  look  for  targets, 
a .  Target  Movement 

Since  some  of  the  live  targets  are  movers,  their 
locations  are  updated  first.  Moving  targets  move  at  a 
fixed  speed  for  a  fixed  duration  in  a  linear  fashion  before 
they  stop  and  change  direction.  The  speed  and  move 
duration  are  both  variable  but  in  this  analysis  they  are 
held  constant  at  27  km/h  and  80  seconds  respectively.  At 
the  end  of  a  target's  move,  a  new  azimuth  is  generated 
randomly  and  the  target  commences  its  movement.  Azimuths 
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that  will  lead  a  target  out  of  the  objective  area  by  the 
end  of  its  move  segment  are  not  allowed.  Figure  8  below 
shows  one  possibility  for  three  consecutive  target  moves. 


Figure  8 .  Target  Movement  Example 


b.  Target  Classification 

Once  target  location  adjustments  are  made  for  the 
movers,  the  list  of  Target  objects  is  iterated  through  to 
determine  which  targets  are  in  the  current  grid  of 

interest.  Targets  that  are  located  in  the  grid  are  pulled 

from  the  master  target  list  and  added  to  a  separate 

candidate  list.  The  candidate  list  is  then  iterated 

through  and  each  target  is  classified  as  live,  dead  or  a 
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decoy  based  on  the  conditional  probability  of 
classification. 

This  is  the  part  of  the  model  where  accuracy 
plays  its  role.  The  probability  of  classification  given  a 
detection  directly  affects  the  quality  of  a  target 
classification.  The  accuracy  levels  are  determined  by  the 
following  sets  of  parameter  settings  provided  by  the 
sponsor : 


Table  4.  Accuracy  Levels 


probability  of  classification  (perceived  |  actual) 

High 

Medium 

Low 

0 

p(live  |  live) 

0.8 

0.6 

0.4 

> 

1 

p(dead  |  live) 

0.1 

0.2 

0.3 

—I 

p(decoy  |  live) 

0.1 

0.2 

0.3 

"O 

p(live  |  dead) 

0.1 

0.2 

0.3 

re 

CD 

p(dead  |  dead) 

0.8 

0.6 

0.4 

Q 

p(decoy  |  dead) 

0.1 

0.2 

0.3 

> 

r\ 

p(live  |  decoy) 

0.1 

0.2 

0.3 

w 

o 

CD 

p(dead  |  decoy) 

0.1 

0.2 

0.3 

Q 

p(decoy  |  decoy) 

0.8 

0.6 

0.6 

For  example,  if  the  current  target  under 

evaluation  is  actually  dead  and  the  p (target  is  perceived 
live  |  target  is  actually  dead)  =  .2,  then  there  is  a  20% 

chance  that  this  target  will  be  misclassif ied  as  live. 
Targets  that  are  classified  as  dead  or  decoy  are  returned 
to  the  master  target  list.  However,  any  target  that  is 

classified  as  live  is  sent  to  the  "Fire  At  Target"  event  as 
one  of  the  parameters.  The  other  parameter  sent  to  the 

"Fire  At  Target"  event  is  the  location  of  the  UAV  when  this 
target  was  detected  and  classified. 

c.  Target  Management 

Once  a  target  is  perceived  (or  classified)  as 

live  it  does  not  go  back  into  the  master  target  list  until 
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later  in  the  process.  The  reason  for  this  is  because  the 
same  UAV  will  make  multiple  glimpses  in  the  same  grid 
before  it  moves  to  the  next  grid.  If  it  has  the 
opportunity  to  reclassify  the  same  target  again  as  live  on 
a  subsequent  glimpse,  then  another  munition  gets  called  in 
on  the  same  target  and  the  overall  number  of  rounds  to  kill 
the  targets  at  the  end  of  the  simulation  becomes  abnormally 
high.  When  the  "Determine  Targets"  event  iterates  through 
the  master  target  list,  if  a  target  is  identified  as  live 
it  will  not  be  available  for  detection  and  classification 
again  until  after  the  round  designated  for  it  has  landed. 

2.  "Fire  At  Target"  Event 

The  "Fire  At  Target"  event  is  simple  in  what  it  does, 
but  it  is  symbolically  very  important.  This  event  does  not 
take  place  until  after  the  processing  time,  which  started 
at  detection,  has  elapsed.  Since  some  of  the  live  targets 
are  movers,  there  is  always  a  chance  that  the  original 
target  that  prompted  the  firing  of  a  round  may  not  be  in 
the  grid  when  the  round  lands. 

This  is  the  part  of  the  model  where  timeliness  plays 
its  role.  The  actual  values  used  for  the  level  of 
timeliness  in  the  model  come  from  a  normal  distribution 
with  parameters  listed  in  the  following  table: 

Table  5.  Timeliness  Levels 


High 

Medium 

Low 

mean 

10  sec 

30  sec 

60  sec 

std  dev 

1  sec 

3  sec 

6  sec 

The  parameters  passed  in  from  the  "Determine  Targets" 

event,  the  target  and  UAV  location, 
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are  simply  carried 


along  and  passed  on  to  the  next  event.  The  "Fire  At 
Target"  event  does  not  do  anything  with  these  parameters. 
The  purpose  of  this  event  is  to  record  the  expenditure  of  a 
munition  and  immediately  schedule  an  "Assess  Target  Damage" 
event.  Technically  there  would  be  a  time  of  flight  for  the 
round  that  would  take  place  after  the  firing  event. 
However,  that  time  is  accounted  for  as  one  of  the 
components  of  the  aggregated  total  processing  time  leading 
up  to  the  "Fire  At  Target"  event.  Therefore  the  "Assess 
Target  Damage"  event  is  immediately  scheduled  with  a  delay 
of  0.0  seconds . 

3.  "Assess  Target  Damage"  Event 

At  this  point  the  munition  that  was  scheduled  to  be 
fired  (when  a  target  was  perceived  live  back  in  the 
"Determine  Targets"  event)  is  now  about  to  land.  The 
target  that  was  passed  in  to  this  event  as  a  parameter  from 
the  "Fire  At  Target"  event  is  now  placed  back  in  the  master 
target  list. 

a .  Target  Movement 

As  in  the  "Determine  Targets"  event,  moving 
target  locations  must  be  updated.  This  happens  right 
before  the  strike  of  the  round  and  right  after  the  target 
triggering  the  firing  event  is  placed  back  in  the  master 
target  list.  This  gives  the  target  that  has  been  held  out 
of  the  list  a  chance  to  update  its  location  before  the 
round  selects  a  target. 

jb.  Target  Selection 

The  UAV  parameter  that  gets  passed  in  to  this 
event  contains  the  grid  location  of  the  UAV  when  the 
original  target  was  detected  and  classified.  The  target 
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list  is  iterated  through  and  a  new  candidate  list  is  built 
consisting  of  targets  that  are  currently  located  in  the 
grid.  The  candidate  list  is  then  iterated  through  in  order 
to  find  the  original  target.  If  the  original  target  is 
found  then  that  is  the  target  the  round  hits.  If  the 
original  target  is  not  found,  but  there  are  other  targets 
in  the  candidate  list,  then  a  target  is  randomly  chosen 
from  the  list  to  be  hit  by  the  round.  Once  a  target  is 
taken  from  the  candidate  list  to  be  hit  by  the  round,  all 
other  targets  are  returned  to  the  master  target  list.  If 
no  targets  are  in  the  candidate  list,  then  the  round 
becomes  wasted. 

There  are  a  few  important  notes  regarding  the 
accuracy  of  the  munition.  As  depicted  in  figure  9,  it  is 
envisioned  that  Objective  Force  units  conducting  beyond 
line  of  sight  (BLOS)  fire  missions  will  be  utilizing  an 
extended-range  precision-guided  munition  effective  out  to 
12  kilometers.16  Since  the  MCS  Company  in  the  simulation 
model  is  conducting  BLOS  fire  missions  at  maximum  ranges 
from  8-12  kilometers,  when  a  round  is  fired  into  a  grid  it 
will  kill  any  target  in  that  grid  with  a  probability  of 
1.0.  This  seems  consistent  with  the  technical  vision  for 
BLOS  munition  capability  in  the  Objective  Force. 


16  National  Defense  Industrial  Association  2001  Munitions  Executive 
Summit  10-12. 


“Precision  Point 
Target  Defeat ” 


One  Shot. 

.  .At  Least  One  Kill" 


NLOS  4-50KM 

Figure  9. 


BLOS  2-1 2km  LOS  0-4Km 

Objective  Force  Fire  Missions 


As  a  convention  in  the  MCSKM,  the  BLOS  round  will 
always  seek  first  the  target  it  was  aimed  at  and  kill  it  if 
present.  However,  if  the  original  target  is  not  in  the 
grid  the  round  will  randomly  choose  another  target  in  the 
same  grid  and  kill  it. 

If  the  target  chosen  by  the  round  is  actually 
already  dead  or  a  decoy,  even  though  it  was  perceived  live, 
that  target  is  simply  returned  to  the  master  target  list 
and  available  for  detection  again.  If  the  target  chosen  by 
the  round  is  actually  live,  then  that  target  is  passed  as  a 
parameter  to  the  "Change  Target  State"  event  with  a  delay 
of  0.0  seconds. 


4. 

"Change 

Target 

State" 

Event 

The 

"Change 

Target 

State" 

event  makes  some  changes  to 

the  Target  object  that  is  passed  in.  First  it  changes  the 
target  state  from  live  to  dead.  Then,  in  case  the  target 
was  a  mover,  it  ensures  that  the  moving  attribute  is  set  to 
false.  Finally,  the  number  of  live  targets  remaining  is 
decreased  by  one.  It  is  at  this  event  that  the  MCSKM 
terminates  once  the  pre-established  threshold  for  the 
number  of  killed  targets  is  met. 
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5. 


MOE  Explanation 


When  it  comes  to  resource  allocation,  there  is  a 
tension  between  effectiveness  and  efficiency.17  The  mission 
must  get  accomplished  so  effectiveness  is  of  primary 
importance.  However,  if  there  are  multiple  ways  of 

accomplishing  the  mission,  the  most  efficient  one  with 
regards  to  expenditure  of  resources  is  preferred. 

By  design,  the  MCSKM  will  eventually  accomplish  the 
mission.  Since  all  targets  are  at  standoff  ranges,  there 
is  no  threat  of  return  fire.  If  the  model  runs  long 
enough,  no  matter  how  bad  the  information  quality  is,  the 
MCS  Company  will  eventually  get  the  enemy  down  to  the 
desired  attrition  level.  Therefore,  the  question  becomes 
how  efficiently  this  can  be  done.  That  is  why  the  number 
of  munitions  fired  is  the  chosen  measure  of  effectiveness 
(MOE)  to  determine  the  relative  importance  of  information 
timeliness,  accuracy,  and  completeness  for  this  simulation. 


17  FM  6-0,  p.  2-26 
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III.  SIMULATION  RESULTS  AND  ANALYSIS 


A.  EXPERIMENTAL  DESIGN 

The  experimental  design  implemented  for  analysis  of 
the  Mounted  Combat  System  Killing  Machine  (MCSKM)  is  a  3k 
factorial  design,  with  k  set  at  3,  meaning  there  are  3 
factors  under  observation  each  at  three  levels.18  In  this 
case,  the  three  factors  are  timeliness,  accuracy  and 
completeness.  The  levels  for  each  factor  are  represented 
by  coded  variables  as  such:  high  =  1,  medium  =  0,  low  =  -1. 
With  three  factors  at  three  levels  each,  there  are  a  total 
of  27  design  points.  The  following  table  displays  the 
complete  design: 

Table  6.  33  Factorial  Experimental  Design 


Design  Point 

Completeness  level 

Accuracy  level 

Timeliness  level 

1 

1 

1 

1 

2 

1 

1 

0 

3 

1 

1 

-1 

4 

1 

0 

1 

5 

1 

0 

0 

6 

1 

0 

-1 

7 

1 

-1 

1 

8 

1 

-1 

0 

9 

1 

-1 

-1 

10 

0 

1 

1 

11 

0 

1 

0 

12 

0 

1 

-1 

13 

0 

0 

1 

14 

0 

0 

0 

15 

0 

0 

-1 

16 

0 

-1 

1 

17 

0 

-1 

0 

18 

0 

-1 

-1 

19 

-1 

1 

1 

20 

-1 

1 

0 

21 

-1 

1 

-1 

22 

-1 

0 

1 

23 

-1 

0 

0 

24 

-1 

0 

-1 

25 

-1 

-1 

1 

26 

-1 

-1 

0 

27 

-1 

-1 

-1 

18  Montgomery,  p.  281 
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The  tables  below  show  specifically  what  values  are 


used  for  each  of  the  three  factors  and  their  three  levels: 


Table  7.  Actual  Values  for  Timeliness 


High(+1) 

Medium(O) 

Low(-I) 

mean 

10  sec 

30  sec 

60  sec 

std  dev 

1  sec 

3  sec 

6  sec 

Table  8.  Actual  Values  for  Completeness 


High(+1) 

Medium(0) 

Low(-I) 

UAV  Speed 

120  km/h 

60  km/h 

30  km/h 

Grid  Size 

400m 

200m 

100m 

Table  9.  Actual  Values  for  Accuracy 


probability  of  classification  (perceived  |  actual) 

High(+1) 

Medium(0) 

Low(-I) 

0) 

p(live  |  live) 

0.8 

0.6 

0.4 

> 

p(dead  |  live) 

0.1 

0.2 

0.3 

_l 

p(decoy  |  live) 

0.1 

0.2 

0.3 

■a 

p(live  |  dead) 

0.1 

0.2 

0.3 

a 

0) 

p(dead  |  dead) 

0.8 

0.6 

0.4 

D 

p(decoy  |  dead) 

0.1 

0.2 

0.3 

> 

r\ 

p(live  |  decoy) 

0.1 

0.2 

0.3 

u 

O 

0) 

p(dead  |  decoy) 

0.1 

0.2 

0.3 

Q 

p(decoy  |  decoy) 

0.8 

0.6 

0.6 

B .  EXPERIMENT  RESULTS 

For  the  run  of  the  full  experiment,  input  parameter 
values  were  used  that  correspond  to  the  three  levels  of 
timeliness,  accuracy,  and  completeness  given  in  the  tables 
above.  The  individual  parameters  were  changed  in  groups, 
as  opposed  to  individually,  based  on  how  each  of  the  three 
main  factors  have  been  defined  for  this  research.  Each 
design  point  represents  a  unique  combination  of  factor 
settings.  The  response  for  each  design  point  represents 
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the  number  of  munitions  fired  from  the  entire  collection  of 
MCS  weapon  platforms  to  kill  80%  of  the  live  targets.  For 
each  design  point  100  replications  of  the  MCSKM  were  run 
and  the  mean  response  for  each  design  point  is  listed  in 
the  table  below: 


Table  10.  Experiment  Summary  with  Means 


Design  Point 

Completeness  level 

Accuracy  level 

Timeliness  level 

Munitions 

1 

1 

1 

1 

71.23 

2 

1 

1 

0 

81.33 

3 

1 

1 

-1 

99.92 

4 

1 

0 

1 

106.38 

5 

1 

0 

0 

112.72 

6 

1 

0 

-1 

133.88 

7 

1 

-1 

1 

143.23 

8 

1 

-1 

0 

156.94 

9 

1 

-1 

-1 

170.57 

10 

0 

1 

1 

81.6 

11 

0 

1 

0 

137.01 

12 

0 

1 

-1 

142.16 

13 

0 

0 

1 

121.23 

14 

0 

0 

0 

188.77 

15 

0 

0 

-1 

204.82 

16 

0 

-1 

1 

161.13 

17 

0 

-1 

0 

251.8 

18 

0 

-1 

-1 

248.87 

19 

-1 

1 

1 

104.42 

20 

-1 

1 

0 

209.87 

21 

-1 

1 

-1 

201.38 

22 

-1 

0 

1 

154.83 

23 

-1 

0 

0 

279.99 

24 

-1 

0 

-1 

284.39 

25 

-1 

-1 

1 

214.07 

26 

-1 

-1 

0 

382.96 

27 

-1 

-1 

-1 

346.58 

1 .  Model  Verification 


At  a  glance,  the  results  seem  to  meet 
sense  expectations.  The  high  level  settings 
timeliness,  accuracy  and  completeness  produce 


some  common- 
for  each  of 
the  best  MOE 
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of  71.23  munitions.  Likewise,  the  low  level  settings  for 
each  of  timeliness,  accuracy  and  completeness  produce 
nearly  the  worst  MOE  of  346.58  munitions.  The  table  below 
shows  the  mean  responses  for  all  factor  levels,  evaluated 
one  factor  at  a  time: 


Table  11.  Mean  Responses  for  each  Factor 


Timeliness 

mean 

1 

128.68 

0 

200.15 

-1 

203.62 

Accuracy 

mean 

1 

125.44 

0 

176.33 

-1 

230.68 

Completeness 

mean 

1 

119.58 

0 

170.82 

-1 

242.05 

Again,  intuition  is  confirmed  by  the  above  results. 
One  would  expect  the  MOE  to  get  worse  as  the  level  settings 
for  each  factor  vary  from  high  to  low.  According  to  Law 
and  Kelton,  an  indicator  that  a  simulation  is  working 
properly  is  that  it  produces  reasonable  output  when  run 
under  a  variety  of  settings  of  the  input  parameters.19 

There  is  ample  evidence  to  suggest  that  the  MCSKM 
works  properly.  As  described  above,  the  results  meet  a 
basic  level  of  validity  as  far  as  the  directional  effects 
one  would  expect  to  see  given  the  different  parameter 
settings.  Also,  a  detailed  trace  on  the  execution  of  the 
model  was  conducted  by  the  author,  stepping  through  the 
MCSKM  event  by  event.  All  locations  were  plotted  by  hand 
and  state  variables  were  tracked  externally  to  the 
simulation.  Finally,  subject  matter  experts  at  TRAC- 
Monterey  concurred  with  the  results  and  agreed  they  were 
consistent  with  the  parameter  settings  chosen. 


19  Law  &  Kelton,  p.  270 
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2. 


Determining  Factor  Significance 


With  a  model  that  has  produced  some  meaningful  output, 
the  task  becomes  determining  the  significance  of  the 
information  quality  components.  How  important  is  each 
factor  and  how  much  does  each  factor  influence  the  number 
of  munitions  fired? 

a.  Analysis  of  Variance 

The  first  step  in  answering  these  questions  is  to 
look  at  the  results  of  a  three-factor  analysis  of  variance 
(ANOVA) .  The  table  below  (computed  in  the  S-PLUS 
statistical  software  package)  displays  timeliness,  accuracy 
and  completeness  as  the  sources  of  variation  along  with  all 
possible  interactions: 


Table  12.  Analysis  of  Variance  for  MCSKM  Munitions 

Fired 


Source  of  Variation 

Degress  of 
Freedom 

Sum  of 
Squares 

Mean 

Square 

F  Value 

P  Value 

Completeness 

2 

6810180 

3405090 

1761.3 

0.0000 

Accuracy 

2 

4986478 

2493239 

1289.6 

0.0000 

Timeliness 

2 

3220931 

1610465 

833.0 

0.0000 

Completeness:  Accuracy 

4 

374358 

93589 

48.4 

0.0000 

Completeness:Timeliness 

4 

1230365 

307591 

159.1 

0.0000 

Accu  racy:  Ti  m  el  i  n  ess 

4 

112753 

28188 

14.6 

0.0000 

Completeness:  Accuracy:  Timeliness 

8 

56096 

7012 

3.6 

0.0003 

Residuals 

2673 

5167669 

1933 

It  can  be  seen  that  all  factors  and  interactions 
are  statistically  significant  even  at  the  a  =  .001  level.  In 
other  words,  all  sources  of  variation  listed  above  affect 
the  number  of  munitions  fired. 

jb.  Polynomial  Regression 

Even  though  all  factors  and  interactions  are 
significant,  there  is  still  no  indication  of  their  relative 
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importance  in  determining  the  number  of  munitions  fired. 
In  order  to  gain  insight  into  this  aspect  of  the  analysis, 
a  complete  second  order  regression  model  was  fit  to  the 
data.  The  two-way  interactions  are  important  in  order  to 
gain  insight  about  how  factor  combinations  perform.  The 
squared  terms  are  important  to  have  in  the  model  in  order 
to  check  for  non-linear  factor  behavior.  The  value  used 
for  each  factor  was  the  number  corresponding  to  the  level 
setting  (1,  0,  -1)  instead  of  the  actual  value  used  in  the 

simulation.  All  replications  were  used  in  building  the 
regression  model.  There  are  27  design  points  and  100 
replications  for  each  design  point  for  a  total  of  2700  data 
points.  The  regression  model  was  fit  using  Excel's 

regression  feature  and  the  results  are  detailed  in  table  13 
below : 
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Table  13. 


SUMMARY  OUTPUT 


Polynomial  Regression  Model  of  MCSKM 
Response 


Regression  Statistics 


Multiple  R 

0.85 

R  Square 

0.73 

Adjusted  R  Square 

0.73 

Standard  Error 

46.88 

Observations 

2700 

AN  OVA 


df 

SS 

MS 

F 

Significance  F 

Regression 

9 

16047867.76 

1783096.42 

811.46 

0.00 

Residual 

2690 

5910962.58 

2197.38 

Total 

2699 

21958830.35 

Coefficients 

Standard  Error 

tStat 

P-value 

Lower  95% 

Upper  95% 

Intercept 

192.34 

2.39 

80.58 

0.00 

187.66 

197.02 

Completeness 

-61.24 

1.10 

-55.43 

0.00 

-63.40 

-59.07 

Accuracy 

-52.62 

1.10 

-47.63 

0.00 

-54.79 

-50.46 

Timeliness 

-37.47 

1.10 

-33.91 

0.00 

-39.64 

-35.30 

Completeness2 

10.00 

1.91 

5.22 

0.00 

6.24 

13.75 

Accuracy2 

1.73 

1.91 

0.90 

0.37 

-2.03 

5.48 

Timeliness2 

-34.01 

1.91 

-1 7.77 

0.00 

-37.76 

-30.25 

Completeness:Accuracy 

17.47 

1.35 

12.91 

0.00 

14.82 

20.13 

Completeness:Timeliness 

22.96 

1.35 

16.97 

0.00 

20.30 

25.61 

Accuracy:Timeliness 

5.12 

1.35 

3.78 

0.00 

2.46 

7.77 

The  intercept  alone  represents  the  predicted 
response  when  all  levels  are  at  their  medium  level  (0)  . 
The  other  terms  in  the  model  come  into  play  when  the  level 
of  a  factor  changes  to  high  (  +  1)  or  low(-l)  .  With  an  R- 
Square  value  of  .731,  this  regression  model  accounts  for  a 
significant  amount  of  the  variation  in  the  MCSKM  output 
data.  The  graphs  below  depict  the  relationship  between  the 
regression  model  predictions  and  the  actual  simulation 
responses  as  well  as  demonstrate  the  constant  variance  in 
the  residuals: 
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regression  model  to  make  some  general  observations  about 
the  way  timeliness,  accuracy,  and  completeness  behave  in 
this  simulation. 

All  terms  in  the  regression  model  are  significant 
at  the  a  =0 . 001  level  except  for  the  [accuracy] “  term.  Not 
only  is  the  p-value  for  [accuracy]2  large,  but  the 
coefficient  is  quite  small  so  it  has  little  impact  on  the 
response.  These  two  things  are  indicators  that  the  effect 
of  accuracy  on  the  response  is  essentially  linear  according 
to  our  coding. 

However,  the  effects  of  timeliness  and 
completeness  are  not  linear.  First,  consider  what  happens 
to  the  response  when  all  factors  are  at  the  medium  level 
and  timeliness  alone  is  varied.  These  changes  are 
reflected  in  table  14  below: 


Table  14.  Changing  Timeliness  Alone  from  Medium 

Level 


Intercept 

Completeness 

Accuracy 

Timeliness  C2 

A2 

T2 

CA 

CT 

AT 

Munitions 

settings: 

0 

0 

1  0 

0 

1 

0 

0 

0 

High 

values: 

192.3 

-37.5 

-34 

=120.8 

settings: 

0 

0 

0  0 

~0~ 

0 

0 

0 

0 

Medium 

values: 

192.3 

=192.3 

settings: 

0 

0 

-1  0 

~0~ 

1 

0 

0 

0 

Low 

values: 

192.3 

+37.5 

-34 

=195.8 

If  timeliness  is  increased  to  its  high  level 
(setting  of  1)  the  number  of  munitions  goes  down  by  71.5 
(-37.5  -  34)  munitions.  But  if  the  level  of  timeliness  is 
decreased  to  its  low  level  (setting  of  -1)  the  number  of 
munitions  goes  up  by  only  3.5  (37.5  -34)  .  This  is  clearly 

not  linear  behavior  and  having  the  squared  term  in  the 
regression  model  captures  this  dynamic.  The  bigger 

resulting  change  from  the  medium  setting  is  in  the 
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direction  of  decreasing  the  number  of  munitions  fired  in 
spite  of  the  fact  that  going  from  the  middle  level  to  the 
high  level  (30  sec  to  10  sec)  is  a  shorter  step  than  going 
from  the  middle  level  to  the  low  level  (30  sec  to  60  sec) . 

On  the  other  hand,  the  same  procedure  applied  to 
the  completeness  factor  indicates  the  opposite  effect. 
Table  15  below  shows  what  happens  when  completeness  levels 
are  changed  in  both  directions  from  the  medium  level: 

Table  15.  Changing  Completeness  Alone  from  Medium 

Level 


Intercept 

Completeness 

Accuracy 

Timeliness 

c2 

A2 

T2 

CA 

CT 

AT 

Munitions 

settings: 

1 

0 

0 

1 

0 

0 

0 

0 

0 

High 

values: 

192.3 

-61.2 

10.0 

=141.1 

settings: 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Medium 

values: 

192.3 

=192.3 

settings: 

-1 

0 

0 

1 

0 

0 

0 

0 

0 

Low 

values: 

192.3 

61.2 

10.0 

=263.5 

If  completeness  is  increased  to  its  high  level 
(setting  of  1)  the  number  of  munitions  goes  down  by  51.2 
(-61.2  +  10.0)  munitions.  But  if  the  level  of  completeness 
is  decreased  to  its  low  level  (setting  of  -1)  the  number  of 
munitions  goes  up  by  71.2  (61.2  +  10.0)  .  This  is  clearly 
not  linear  behavior.  However,  the  bigger  change  from  the 
medium  level  setting  is  in  the  direction  of  increasing  the 
number  of  munitions  fired  in  spite  of  the  fact  that  going 
from  the  middle  level  to  the  low  level  200m  grid  to  100m 
grid)  is  a  shorter  step  than  going  from  the  middle  level  to 
the  high  level  (200m  grid  to  400m  grid) . 

By  contrast,  since  the  [accuracy] _  term 
coefficient  is  so  small  (1.7),  it  would  have  little  effect 
on  the  linearity  of  accuracy  if  we  applied  the  same 
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procedure  again.  The  response  would  essentially  change  by 
+  52.6  rounds,  the  accuracy  term  coefficient,  which  is 

linear  behavior. 

These  findings  can  help  prioritize  the 
expenditure  of  resources  based  on  different  goals.  If  the 
intent  is  to  guard  against  losing  capability,  the  area  of 
information  completeness  should  be  a  maintenance  priority 
since  this  analysis  suggests  a  small  drop  in  this  factor 
level  translates  into  accelerated  degradation  in  munition 
expenditure  efficiency.  However,  if  the  intent  is  to 
increase  the  current  capability,  the  area  of  information 
timeliness  should  be  a  research  and  development  priority 
since  this  analysis  suggests  a  small  increase  in  this 
factor  level  translates  into  accelerated  improvement  in 
munition  expenditure  efficiency. 

c.  Correlation  and  R-Square  Contribution 

As  mentioned  earlier,  the  polynomial  regression 
model  accounts  for  73%  of  the  variability  in  the  MCSKM 
output  based  on  the  R-Square  value.  If  we  look  at  the 
component  breakdown  of  this  73%  by  how  much  each  term  in 
the  regression  equation  contributes,  we  can  obtain  an 
indication  of  the  relative  importance  of  timeliness, 
accuracy,  and  completeness. 

In  a  designed  experiment  like  this  one,  the 
coefficients  of  the  terms  in  the  regression  model  will  be 
uncorrelated  (as  displayed  in  figure  11)  .  Therefore,  we 
can  actually  compute  the  specific  amount  of  the  total  R- 
Square  value  for  which  each  term  is  responsible.20 


20  Neter 
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The  first  step  is  to  compute  the  correlation  of 
munitions  expended  with  every  other  term  in  the  model. 
This  can  be  done  by  extracting  the  last  row  of  the 
correlation  matrix  produced  by  Excel: 


Completeness 

Accuracy 

Timeliness 

c2 

A2 

T2 

CA 

CT  AT  Munitions 

Completeness 

1 

Accuracy 

0 

1 

Timeliness 

0 

0 

1 

Completeness2 

5E-18 

-8E-18 

-4E-19 

1 

Accuracy2 

-8E-18 

-8E-18 

-4E-19 

-3E-18 

1 

Timeliness2 

-4E-19 

-4E-19 

-4E-19 

-2E-17 

-2E-17 

1 

Completeness:Accuracy 

0 

0 

0 

-7E-18 

-7E-18 

5E-19 

1 

Completeness  Timeliness 

0 

0 

0 

5E-19 

5E-19 

5E-19 

0 

1 

Accuracy:Timeliness 

0 

0 

0 

5E-19 

5E-19 

5E-19 

0 

0  1 

Munitions 

-0.554 

-0.476 

-0.339 

0.052 

0.009 

-0.178 

0.129 

0.170  0.038  1 

Figure  12 . 

Correlation 

Matrix 

of 

Regression 

Model 

Inputs 


The  square  of  the  correlation  between  Munitions 
and  each  term  in  the  regression  model  becomes  that  term' s 
component  contribution  to  the  total  R-Square.  Table  16  and 
figure  13  below  summarize  this  relationship: 


Table  16.  Component  Contribution  to  Total  R-Square 

Regression  Input:  Completeness  Accuracy  Timeliness  C2  A2  T2  CA  CT  AT 

Correlation  with  Munitions:  -0.5544  -0.4764  -0.3392  0.0522  0.0090  -0.1778  0.1292  0.1697  0.0378 

R-Square  Contribution:  0.3074  0.2270  0.1151  0.0027  0.0001  0.0316  0.0167  0.0288  0.0014 

Total  R-Square:  0.7308  (sum  of  above  row) 
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R-Square  Contribution 


Polynomial  Regression  Term 


Figure  13.  Regression  Term  R-Square  Contribution 

Chart 

Table  17  below  displays  the  percentage  of  the 
total  R-Square  for  which  each  term  is  responsible.  The 
term  that  explains  the  most  variability  in  the  number  of 
munitions  fired  is  completeness  at  42%.  Accuracy  and 

timeliness  follow  at  31%  and  16%  respectively. 

[Timeliness ]  -  as  well  as  the  two  interactions  of 
completeness : accuracy  and  completeness : timeliness  explain 
roughly  2-4%  of  the  variability  each. 

Table  17.  Percentage  of  Total  R-Square  Contribution 

Regression  Input:  Completeness  Accuracy  Timeliness  C2  A2  T2  CA  CT  AT 

R-Square  Contribution:  0.3074  0.2270  0.1151  0.0027  0.0001  0.0316  0.0167  0.0288  0.0014 

Percentage  of  Total  R-Square:  42.06%  31.06%  15.75%  0.37%  0.01%  4.32%  2.28%  3.94%  0.20% 
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3. 


Summary 


The  analysis  above  of  the  MCSKM  output  shows  several 
things.  From  the  analysis  of  variance  it  can  be  seen  that 
the  three  factors  of  timeliness,  accuracy  and  completeness 
as  well  as  their  two-way  interactions  are  significant.  In 
other  words,  each  factor  has  a  unique  impact  on  the  number 
of  munitions  fired.  No  two  factors  are  interchangeable. 

Building  a  complete  second  order  regression  model  that 
fits  the  MCSKM  output  reasonably  well  provides  a  framework 
to  look  at  the  relative  significance  of  the  terms  used  to 
build  the  model.  The  coefficients  on  the  squared  terms 
provide  an  indication  of  the  linearity  of  the  three 
factors.  The  size  of  the  coefficient  on  the  squared  term 
provides  an  indication  of  the  degree  of  non-linearity  that 
exists  with  regards  to  that  factor.  The  sign  of  the 
coefficient  is  an  indicator  of  which  direction  of  travel 
from  the  medium  setting  provides  the  bigger  change  in  the 
number  of  munitions.  A  positive  sign  on  the  coefficient  of 
the  squared  term  indicates  that  the  number  of  rounds 
changes  more  as  the  level  of  the  main  factor  goes  down.  A 
negative  sign  on  the  coefficient  of  the  squared  term 
indicates  that  the  number  of  rounds  changes  more  as  the 
level  of  the  main  factor  goes  up. 

The  regression  model  also  indicates  there  exists  a 
beneficial  timeliness : accuracy  interaction  when  each  of 
these  factors  is  set  at  its  high  level.  Figure  14  below 
demonstrates  that  the  model's  top  three  predictions  all 
occur  when  timeliness  =  accuracy  =  +1. 
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Figure  14.  Regression  Model  Top  Predictions 

A  beneficial  timeliness : accuracy  interaction  is 
further  evidenced  by  the  fact  that  three  of  the  top  five 
MOE  values  resulting  from  the  actual  simulation  runs  (refer 
back  to  table  10)  are  at  design  points  1,  10,  and  19,  where 
timeliness  and  accuracy  are  each  at  their  high  level. 
Therefore,  while  at  their  high  levels,  the  interaction  of 
timeliness  and  accuracy  negates  the  contribution  of 
completeness . 

Finally,  the  correlation  of  Munitions  to  the  other 
terms  in  the  regression  equation  provides  a  way  to  get  at 
the  component  pieces  of  the  total  R-Square  value.  This 
gives  a  good  indication  of  the  impact  of  each  term' s 
influence  on  the  number  of  munitions  fired. 
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IV.  CONCLUSIONS 


A.  INFORMATION  QUALITY  COMPONENT  IMPACT 


The  goal  of 

this 

thesis 

was  to  draw  some  broad 

conclusions 

about 

how 

the 

individual 

components 

of 

information 

quality  can 

influence  combat 

outcomes . 

It 

turns  out 

that 

information 

timeliness , 

accuracy 

and 

completeness  each  have  a  distinctive  and  significant  impact 
on  the  results  of  a  combat  scenario. 

1.  Simulation  Results  Implications 

In  this  scenario  we  were  able  to  see  that  varying  the 
levels  of  these  components  affect  the  number  of  munitions 
required  to  kill  a  given  percentage  of  enemy  targets. 
Although  the  output  of  the  MCSKM  is  heavily  dependent  upon 
the  scenario,  the  MOE,  and  input  data,  we  discovered  that 
that  the  individual  effect  of  timeliness,  accuracy  and 
completeness  may  not  be  linear.  Knowing  where  and  how  to 
achieve  an  accelerated  return  based  on  an  incremental 
change  to  any  of  these  components  is  important.  We  also 
discovered  in  this  analysis  that  there  are  significant 
synergistic  effects  that  take  place  between  information 
components.  Knowing  that  the  combined  effects  of  two  of 
these  components  can  overshadow  the  effect  of  the  remaining 
component  is  important  as  well. 

The  dynamic  relationship  among  information  quality 
components  that  emerged  from  this  analysis  is  likely  to 
exist  in  virtually  any  given  scenario  and  the  particulars 
of  that  relationship  will  be  unique  to  that  scenario.  This 
information  in  the  hands  of  a  concept  developer  allows  him 
to  make  wise  choices  in  determining  what  technologies  and 
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tactics  are  needed  to  improve  the  success  of  units 

optimized  for  specific  missions. 

2.  Insights  Gained  from  Preliminary  Experimentation 

Although  the  MCSKM  has  the  ability  to  represent  a  wide 
variety  of  parameters,  many  were  held  fixed  in  this 
analysis.  The  fact  that  many  of  the  parameters  were  not 
varied  suggests  much  future  work  that  will  be  addressed  in 
the  next  section.  However,  below  are  just  a  few  insights 
gained  from  trial  and  error: 

a.  Magnification  of  Small  Performance  Errors 

With  a  probability  of  false  detection  and  a 
probability  of  detection  being  varied  as  part  of  the 
accuracy  component,  the  round  counts  were  ranging  from 

approximately  300  with  factors  at  the  high  levels  to  14,000 
with  factors  at  the  low  levels.  The  UAV  can  make  a  false 
detection  at  every  glimpse.  In  the  course  of  an  entire  run 
of  a  scenario  there  are  so  many  glimpses  that  even  if  the 
probability  of  false  detection  is  as  small  as  .01  there 
could  be  hundreds  of  false  detections  each  resulting  in  a 
wasted  round. 

The  probability  of  detection  compounded  this 
problem  by  dragging  out  the  simulation.  If  a  target  was 

present  in  the  grid  but  not  detected,  the  UAV  would  pass 

over  it  and  have  to  randomly  come  back  to  it  at  a  later 
time.  By  the  time  the  UAV  comes  back  to  the  target  it  has 
had  numerous  opportunities  to  make  false  detections, 
misclassify  dead  or  decoy  targets  as  live,  and  waste  more 
rounds . 

The  attrition  level  was  yet  another  contributor 
to  the  problem.  After  the  majority  of  the  live  targets  are 
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found  and  killed,  the  UAV  has  to  keep  looking  for  the  last 
few  live  targets  and  spends  a  lot  of  time  wasting  rounds  in 
the  meantime. 


In  order  to  keep  the  number  of  munitions  at  a 
reasonable  level,  reduction  of  the  complexity  of  the 
accuracy  component  was  necessary.  This  was  achieved  by 
taking  the  effect  of  false  targets  and  the  probability  of 
detection  out  of  the  scenario.  The  probability  of 

classification  alone  produced  more  interpretable  results. 

After  the  accuracy  component  was  brought  under 
control,  the  attrition  level  was  less  influential  in  high 
round  counts.  However,  keeping  the  attrition  level  at  80% 
provided  a  stopping  criteria  that  allowed  the  simulation  to 
run  in  a  reasonable  amount  of  time  (which  is  important  for 
multiple  runs)  .  The  scenario  chosen  for  this  analysis 
suggested  the  80%  attrition  factor. 

The  insight  gained  from  all  of  this  was  that 
imperfect  information  results  in  substantial  inefficiencies 
in  destroying  targets  and  even  small  performance  errors 
become  magnified  over  the  course  of  a  lengthy  engagement. 

b.  Completeness  of  Information  Depends  on  Time 

Initially  the  completeness  component  of 
information  was  modeled  strictly  by  the  size  of  the  grid 
square  representing  the  footprint  of  the  UAV' s  sensor. 
However,  the  results  from  these  simulation  runs  did  not 
make  much  intuitive  sense.  It  became  apparent  that  a  UAV 
could  look  at  four  100m  by  100m  grid  squares  in  the  same 
amount  of  time  it  could  look  at  one  200m  by  200m  grid 
square.  This  happened  because  the  UAV  traveled  at  a  fixed 
speed  and  the  time  in  the  grid  square  was  adjusted  at  each 
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completeness  level  to  account  for  this.  In  other  words, 
for  any  given  block  of  time  the  same  amount  of  area  on  the 
ground  was  potentially  covered  regardless  of  the 
completeness  setting. 

As  a  reminder,  completeness  describes  the  level 
to  which  all  the  relevant  items  of  information  are 
available.  Since  target  information  was  the  relevant  item 
in  this  scenario,  the  piece  of  information  that  contained 
data  on  the  most  number  of  targets  was  the  most  complete. 
To  model  completeness  more  appropriately,  the  levels  were 
redefined  so  that  time  in  the  grid  square  was  held  fixed 
and  the  size  of  the  grid  square  changed.  This  required  UAV 
speed  (which  was  previously  held  constant)  to  vary  in 
conjunction  with  the  grid  size. 

B.  FUTURE  STUDY  RECOMMENDATIONS 

A  more  comprehensive  understanding  of  the  impact  of 
information  on  combat  outcomes  requires  further  research  in 
several  areas.  Some  logical  ways  to  proceed  from  this 
research  are  listed  below. 

1.  Refine  the  Parameters  Associated  with  Timeliness, 
Accuracy  and  Completeness 

It  was  difficult  to  come  up  with  just  the  right  way  to 
model  the  components  of  information  quality.  There  are 
other  variables  that  could  be  associated  with  each 
component.  Choosing  these  variables,  as  well  as  the 
appropriate  levels  for  each,  and  then  relating  them  the 
proper  way  would  improve  the  quality  of  the  response  and 
provide  further  insights  into  the  dynamics  of  how  these 
information  quality  components  relate  to  each  other  and  to 
the  combat  outcome. 
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2. 


Apply  Analysis  Framework  to  Other  Scenarios 


The  MCSKM  is  adaptable  to  explore  many  other 
scenarios.  This  can  be  done  by  modifying  the  objective 
area  size  and  shape,  number  of  UAVs  used,  distributions 
used  for  the  varying  parameters,  number  and  types  of 
targets,  and  input  values  for  parameters.  Scenarios  could 
be  compared  with  one  another  to  make  observations  about  how 
the  relationship  among  timeliness,  accuracy  and 
completeness  may  differ. 

3 .  Multiple  UAV  Types 

Although  the  MCSKM  is  currently  capable  of  portraying 
a  variable  number  of  UAVs,  they  all  have  the  same 
characteristics.  Modifying  the  MCSKM  to  allow  for  multiple 
types  of  UAVs  would  facilitate  the  exploration  of  a  wider 
variety  of  scenarios  and  provide  interoperability  with 
existing  and  future  sensor  mix  optimization  models. 

4 .  Complementary  Study  on  the  Value  of  Information 

Information  can  be  broken  down  into  two  attributes: 
value  and  quality.  The  focus  of  this  thesis  was  on 
information  quality  in  terms  of  timeliness,  accuracy  and 
completeness.  A  study  on  the  value  of  information  would 
provide  additional  insights  into  how  information  affects 
combat  outcomes  and,  combined  with  this  study,  provide  a 
more  consummate  interpretation  of  the  overall  impact  of 
information . 

5 .  Spreadsheet  Version  of  MCSKM 

Although  the  MCSKM  is  written  in  Java  as  a  discrete- 
event  simulation,  a  spreadsheet  implementation  of  the  basic 
concepts  behind  the  MCSKM  does  exist.  The  spreadsheet 
version  provides  an  easier  and  more  familiar  environment 
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for  anyone  interested  in  examining  the  underlying 
relationships  in  the  MCSKM.  However,  the  spreadsheet 
version  in  its  current  state  lacks  several  parameters  found 
in  the  MCSKM  and  is  not  generalized  enough  to  make  it 
extensible  for  other  scenarios. 

The  Java  source  code  for  the  MCSKM  and  the  Excel  file 
containing  the  spreadsheet  implementation  discussed  above 
are  both  available  by  contacting  MAJ  Joseph  Baird,  United 
States  Military  Academy,  Department  of  Mathematical 
Sciences,  West  Point,  New  York. 
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